fix(cloudflare): prewarm the uploaded Worker version safely#2593
fix(cloudflare): prewarm the uploaded Worker version safely#2593NathanDrake2406 wants to merge 25 commits into
Conversation
commit: |
Performance benchmarksCompared 0 improved · 1 regressed · 5 within ±1.5%
View detailed results and traces 🟢 improvement · 🔴 regression · ⚫ change below 1.5% · paired base/head |
…fore promoting Workers Cache is keyed per Worker version by default. `--experimental-warm-cdn-cache` issued warmup requests against production before confirming which version answered them, so a request routed to the still-live previous version silently warmed the wrong cache partition. The deploy then promoted the new version anyway, reporting success while the cache stayed cold for real traffic. The fix enforces one invariant: a warmup only counts as successful when the response proves it came from the uploaded Worker version. - `vinext init` provisions a `version_metadata` binding under a fixed name (VINEXT_VERSION_METADATA); the Worker entry reads the current version ID from it and stamps every response with `x-vinext-worker-version`. - Deploy stages the uploaded version at 0% traffic, warms build-discovered paths through a `Cloudflare-Workers-Version-Overrides` header, and rejects any response whose version header doesn't match the uploaded version — without following redirects, so the canonical cache key is what gets verified. - Promotion only happens after warmup completes; in strict mode a failed or unverified warmup aborts before promotion, leaving the previous version at 100% and the new one staged at 0% — already the safe state, so no compensating mutation is issued. In non-strict mode the deploy promotes anyway but does not claim the cache was warmed. - Static exports (`output: "export"`) skip verified warmup entirely: Cloudflare Assets serves them without invoking the Worker, so there's no version header to verify against. Scope cut from the original draft: dropped the generic version-traffic reconciliation module (ambiguous-failure classification, auto-rollback, manual recovery command rendering), the configurable binding-name companion variable in favor of the fixed binding name the runtime always reads, and cross_version_cache config parsing/validation. None of those are required to enforce the core invariant, and each is a separable concern better reviewed on its own. Tests: tests/worker-version.test.ts, tests/cloudflare-cdn-warm.test.ts, tests/cloudflare-cdn-warm-deploy.test.ts, tests/deploy.test.ts, tests/init.test.ts, tests/deploy-prerender-config.test.ts.
4c90d59 to
094c4a9
Compare
…TPR's config reader, and stop reporting unconfirmed warm-ups as plain success
cdn-warm-deployment.ts read Worker name, production host, and legacy_env
suffixing directly out of tpr.ts's WranglerConfig/env map. tpr.ts owns
TPR-specific config extraction; CDN warmup's env-fallback and
Worker-name-suffixing rules do not belong to that module, and the coupling
made tpr.ts a de facto shared deployment-semantics resolver.
Separately, when non-strict warmup exhausted its retries without ever
confirming the uploaded version served a response, deployWithCdnWarmup still
promoted and returned a bare URL string. deploy.ts printed a generic
"Deployed to: <url>" box identical to a fully-warmed deploy, so an operator
had no way to tell a confirmed warm-up from a silent fallback short of
scrolling back through mid-deploy warnings.
Added wrangler-deployment-target.ts to resolve {workerName, productionHost,
versionMetadataBinding} on top of the existing parseWranglerConfig output,
so cdn-warm-deployment.ts no longer touches TOML/env-map shape directly.
deployWithCdnWarmup now returns {url, warmed}; deploy.ts's closing summary
states plainly when a promoted version's cache was not confirmed
pre-warmed, and a console.warn now fires at the retry-exhausted fallback
site that previously logged nothing beyond per-path failures.
Also named the decisions that were previously inlined at effectful
call-sites (promotionPhaseFor, isFullyWarmed, pickDeployedUrl in
cdn-warm-deployment.ts; describeVersionMismatch in cdn-warm.ts) so they're
independently readable and testable from the wrangler/fetch calls around
them.
Added a test covering the previously-uncovered non-strict-promotes-without-
confirmed-warmup path. vp check clean; targeted suite 480/480.
…ing config mid-warm-up resolveCdnWarmupTargetUrl called resolveWranglerDeploymentTarget a second time inside warmAndPromote's warm attempt, reparsing wrangler.jsonc/.toml from disk even though validateCdnWarmupConfiguration had already resolved the same target moments earlier in the same deploy. The duplicate I/O made the WranglerDeploymentTarget abstraction ceremonial rather than real: two call sites could theoretically observe different config state despite describing one deploy. Deleted resolveCdnWarmupTargetUrl. deployWithCdnWarmup now resolves the target once and threads it through; the production-host fallback (`target.productionHost ? https://... : staged.deployedUrl`) is inlined where it's used. Migrated its direct tests in deploy.test.ts onto resolveWranglerDeploymentTarget's `.productionHost` field, which is what they were actually exercising — the deployedUrl fallback itself stays covered by the existing end-to-end warmAndPromote tests. Also split deployWithCdnWarmup into promoteWithoutWarmup and warmAndPromote: they are genuinely different deployment modes (no safe staging split vs. staged-warm-promote) with different guarantees, not just a long function that happened to have an early return. Removed isFullyWarmed — it restated `result.failed === 0` without adding a decision worth naming separately. vp check clean; targeted suite green (deploy.test.ts, cdn-warm-deploy, deploy-prerender-config, cdn-warm, version-deploy, worker-version, init).
getZeroPercentStagingTraffic didn't check the uploaded version ID against the currently-live one. If upload ever returned an ID matching the version already at 100%, it built a traffic split with the same version listed twice (v@100% v@0%) and handed that straight to `wrangler versions deploy`. Added the equality check so that case falls back to promoteWithoutWarmup like any other unsafe traffic shape, instead of reaching wrangler with a malformed split. Also stopped re-deriving previousVersionId from currentVersions[0] in deployWithCdnWarmup — that index is only safe because a non-null stagingTraffic already proved current has exactly one entry, and that proof lived in a different function. Read it from stagingTraffic[0] instead, the value that was actually checked.
Upstream moved TPR KV key construction to the shared key-space helper while this branch added Wrangler deployment-target parsing in the same import block. Keep both APIs so TPR uses the current cache key contract without dropping CDN warmup config support.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4926519547
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Generated environment-specific Wrangler configs flatten the selected environment, failed strict warmups leave zero-percent versions behind, and direct fallback promotion can lose trigger recovery context. That can reject a valid retry or obscure the deployed state after a trigger failure. The deployment flow assumed selected environments always remained nested and that every version in a retry status had positive traffic. Recognize generated configs through Wrangler's targetEnvironment marker, replace stale zero-percent versions when staging a fresh attempt, reuse post-promotion trigger diagnostics, and accept trailing TOML route comments when resolving the production host.
|
/bigbonk review for issues |
|
UnknownError: ProviderInitError |
|
@james-elicx Bonk workflow failed. Check the logs for details. View workflow run · To retry, trigger Bonk again. |
|
/bigbonk review for issues |
1 similar comment
|
/bigbonk review for issues |
|
UnknownError: ProviderInitError |
|
/bigbonk review for issues |
|
UnknownError: "Overloaded" |
|
/bigbonk review for issues |
There was a problem hiding this comment.
Review: staged CDN warmup deployment
I reviewed the full diff and ran the affected test files locally — tests/worker-version.test.ts, tests/cloudflare-cdn-warm.test.ts, tests/cloudflare-cdn-warm-deploy.test.ts, tests/deploy.test.ts, tests/init.test.ts, and tests/deploy-prerender-config.test.ts all pass (474 tests).
The overall design is solid. The core insight — that a 2xx/3xx response alone can't prove the uploaded version served it, because Workers Cache partitions by version ID — is correct, and verifying via x-vinext-worker-version + Cloudflare-Workers-Version-Overrides is the right structural fix. The staging-at-0% / promote-after-warm ordering, the fail-closed strict mode, and the retry-after-stale-0%-version recovery are all handled well and well-tested. The Codex P1/P2 items appear genuinely addressed in e22bc00.
Dev/prod parity check: fetch-handler (the recommended default entry) resolves to app-router-entry/pages-router-entry via virtual:vinext-worker-entry, and custom workers that delegate to handler.fetch() inherit the stamp, so stampWorkerVersion coverage is complete for Worker-served responses. Good.
A few non-blocking observations below.
|
UnknownError: ProviderInitError |
|
/bigbonk review for issues |
|
UnknownError: ProviderInitError |
|
/bigbonk review for issues |
There was a problem hiding this comment.
Review: staged CDN warmup deployment
I read the full diff and the changed source (cdn-warm-deployment.ts, cdn-warm.ts, worker-version.ts, wrangler-deployment-target.ts, utils/toml.ts, tpr.ts, deploy.ts, init-cloudflare.ts, entry stamping) and ran tests/worker-version.test.ts + tests/cloudflare-cdn-warm.test.ts locally (27 pass).
Overall: the design is correct and I found no blocking issues. The structural insight holds: because Workers Cache partitions its key by invoked version ID (unless cross_version_cache), a plain 2xx/3xx cannot prove the uploaded version populated the cache. Verifying via x-vinext-worker-version + a Cloudflare-Workers-Version-Overrides override, with redirect: "manual" so a redirect is proven at its own cache key, is the right fix. The stage-at-0% → warm → promote → triggers-after-promotion ordering keeps the risky window in the already-safe state (previous @100%, new @0%), and strict mode fails closed by leaving the split rather than issuing an undo mutation. getZeroPercentStagingTraffic correctly rejects partial splits, drops stale 0% leftovers, and fails closed when the upload is already at 100%. The path-scoped-route carve-out (hasProductionRoute && !productionHost) is correct, and the custom_domains fallback now applies the same host-wide guard as routes.
Parity confirmed: stampWorkerVersion runs in both app-router-entry and pages-router-entry, and fetch-handler resolves to those, so all Worker-served responses carry the header. The prior Codex P1/P2 and the many /bigbonk rounds all appear genuinely addressed.
Two small, non-blocking observations below.
|
fatal: could not read Username for 'https://github.com': No such device or address |
|
/bigbonk review for issues |
There was a problem hiding this comment.
Review: staged CDN warmup deployment
I read the full diff and the new/changed source (cdn-warm-deployment.ts, cdn-warm.ts, worker-version.ts, wrangler-deployment-target.ts, utils/toml.ts, tpr.ts, deploy.ts, init-cloudflare.ts, entry stamping) and cross-checked the version-metadata binding shape and the version-override header format.
Overall: the design is correct and I found no blocking issues. The structural fix is right — because Workers Cache partitions its key by invoked version ID (unless cross_version_cache), a plain 2xx/3xx can't prove the uploaded version populated the cache, so verifying via x-vinext-worker-version + a Cloudflare-Workers-Version-Overrides override (with redirect: "manual" so a redirect proves at its own cache key) is the correct approach. The stage-at-0% -> warm -> promote -> triggers-after-promotion ordering keeps the risky window in the already-safe state, strict mode fails closed by leaving the split, and getZeroPercentStagingTraffic correctly rejects partial splits / drops stale 0% leftovers / fails closed when the upload is already 100%. stampWorkerVersion parity is complete: both app-router-entry and pages-router-entry stamp at the top-level fetch, and fetch-handler resolves to those. The .id field read from the version-metadata binding matches Cloudflare's binding shape.
The prior Codex P1/P2 and the many /bigbonk rounds all appear genuinely addressed. Everything below is non-blocking.
|
fatal: could not read Username for 'https://github.com': No such device or address |
|
/bigbonk review for issues |
Independent review: staged CDN warmup deploymentI reviewed the full diff and read the new/changed source ( Overall: the design is correct and I found no blocking issues. The structural insight is sound — because Workers Cache partitions its key by invoked version ID (unless Things I verified independently:
The prior Codex P1/P2 and the many
|
|
fatal: could not read Username for 'https://github.com': No such device or address |
Strict CDN warmup errors named the first failure but did not make the verified path count obvious after deployment wrapped the error with its safe-state message. This made truncated CI logs harder to diagnose.\n\nInclude the warmed path count alongside the failure count in the original error so the deployment wrapper preserves both values. Extend the strict staged-deployment regression test to assert the diagnostic and the existing safe-state guidance.
|
/bigbonk review for issues |
Review: staged CDN warmup deploymentI read the full diff and the changed source ( Overall: the design is correct and I found no blocking issues. The structural fix is right — because Workers Cache partitions its key by invoked version ID (unless The prior Codex P1/P2 items and the many Two small, non-blocking observations — both diagnostics/edge polish, no change required to merge:
|
|
fatal: could not read Username for 'https://github.com': No such device or address |
…naming them When CDN warmup is enabled, init silently replaces any existing version_metadata binding with VINEXT_VERSION_METADATA, top-level and in every named environment. The binding name is the property user code reads off env, so a project with a custom Worker entrypoint reading e.g. env.CF_VERSION_METADATA keeps deploying but its worker breaks at runtime. The mistaken assumption was that the binding is vinext-owned config; Workers allows arbitrary names and permits only one version_metadata binding per scope, so an existing differently named binding is a real conflict with user code, not stale config. Init now fails fast on a conflicting binding with a migration message (rename the binding or re-run init without warmup) before mutating any project file, and still adds the binding additively where none exists.
…h warmed The staged deployment counted a path as warmed once the uploaded version answered with its version header and a sub-400 status. That proves the right Worker produced the response, not that Workers Cache stored it: a per-entrypoint cache override or a response-level bypass (Set-Cookie, Cache-Control: no-store/private) returns a healthy 200 the cache never writes, and the deploy summary still printed "pre-warmed and confirmed". The violated invariant is that "the uploaded version produced a 200" and "the edge stored that response in the version's cache partition" are independent facts; only the first was checked. Warmup requests in the deployment transaction now demand cache proof via cf-cache-status: a cache-served status (HIT/STALE/UPDATING/REVALIDATED) counts immediately, a MISS/EXPIRED fill is confirmed by a second identical request that must come back cache-served from the expected version, and BYPASS/DYNAMIC or a missing header fails the path immediately since those are deterministic for the response shape. Standalone warmCdnCache callers keep the old semantics unless they opt in through the new confirmCache option. Tests cover each cf-cache-status branch and assert end-to-end that a producer-valid BYPASS deploy reports warmed: false.
…aded version warmAndPromote promoted the uploaded version to 100% using only the deployment status read before staging. If another deploy promoted its own version during the warmup window, this deploy's version overrides stopped resolving (overrides only apply to versions in the current deployment), non-strict mode recorded the resulting mismatches as ordinary warmup degradation, and the unconditional promotion silently overwrote the other actor's deployment. The violated invariant is ownership: promotion mutates whatever deployment is currently active, and a warmup degradation never grants permission to overwrite a deployment this transaction did not stage. Promotion is now preceded by a fresh `wrangler deployments status` read that must show exactly the staged split (previous@100%, uploaded@0%, order-insensitive). Any other state, or a failed re-read, aborts without promoting and reports the observed traffic split. The check applies in strict and non-strict mode alike. Wrangler exposes no compare-and-swap, so a race remains between this read and the promote command; the check shrinks the exposed window from the whole warmup duration to that gap.
parseWranglerConfig and its WranglerConfig projection lived in tpr.ts while serving both TPR and CDN warmup target resolution, so wrangler-deployment-target.ts imported general deployment-config concerns (cache settings, warmup hosts, environments, version-metadata bindings) from an experimental feature module. That dependency direction invites future deploy features to attach themselves to tpr.ts. Mechanical move of the parser, its helpers, and the projected types into wrangler-config.ts, with no parser behaviour change. tpr.ts consumes parseWranglerConfig for its account/zone/KV fields; wrangler-deployment-target.ts consumes the cache/route/environment/ Worker-name/metadata fields; utils/toml.ts stays a format-level helper. WranglerEnvironmentConfig is no longer exported — nothing outside the parser referenced it.
|
Four hardening commits: 43b43ca makes "warmed" mean stored in cache, not just served; e623639 re-verifies deployment ownership before promotion; c13d76d stops init from renaming a user-owned binding; c7572ab moves Wrangler config parsing out of Warmed means stored, not just served (43b43ca)"The uploaded version produced a 200" and "the edge stored that response in the version's cache partition" are independent facts; the transaction checked only the first. A per-entrypoint cache override or a response-level bypass ( Warmup requests in the deployment transaction now require
The edge's own cache status is the authority, which spares vinext from modelling Wrangler cache configuration or bypass rules locally. Standalone Promotion re-verifies ownership of the deployment (e623639)Promotion trusted the status read taken before staging. If another deploy promoted its own version during the warmup window, this deploy's version overrides stopped resolving (overrides apply only to versions in the current deployment), non-strict mode filed the resulting mismatches under ordinary warmup degradation, and the unconditional promotion silently overwrote the other actor's deployment.
Wrangler exposes no compare-and-swap, so a race remains between the final read and the promote command; the re-check shrinks the exposed window from the whole warmup duration to that gap. Tests cover the concurrent-deploy abort (another version at 100% mid-warmup rejects with zero promote or trigger commands, in non-strict mode) and the failed-re-read abort, and still assert that no reconciling status read happens after a failed promotion. Init refuses to rename a user-owned binding (c13d76d)With warmup enabled, init replaced any existing Init now fails on a conflicting binding before mutating any project file, with instructions to rename the binding or re-run init without warmup. Where no binding exists, init still adds one. Injecting the user's own binding name into the runtime entry at build time would avoid the conflict entirely; that is deferred because it needs build-time plumbing for a situation the fail-fast already surfaces, and it can be added later without changing this behaviour. Wrangler config parsing moved out of tpr.ts (c7572ab)
|
…med pre-warm Warmup previously resolved a single production host: the config parser kept only the first eligible host-wide route or Custom Domain, and warmAndPromote warmed every path against that one origin. The hostname is part of Cloudflare's cache key, so when several hosts are attached to the Worker (for example apex and www routes), each host owns its own version-isolated cache partition. Every non-first host was promoted cold while the deploy summary still claimed "pre-warmed and confirmed", and strict mode succeeded on the same false guarantee. Represent the target as the full set of cache-key origins instead: the parser collects every eligible host-wide origin from route, routes, and custom_domains (deduped, JSONC and TOML alike), the deployment target exposes productionHosts, and the warm transaction covers every origin and path combination. The deployment is reported warmed only when all of them are confirmed; strict mode fails before promotion when any origin cannot be. Repeated TOML [[env.<name>.routes]] blocks previously kept only the last block's host and now accumulate into the same origin set.
… warmed Warmup target resolution kept only routes reducible to a concrete host-wide origin. A path-scoped or wildcard-host route alongside a warmable one was silently dropped: productionHosts stayed non-empty, every path confirmed on the supported origin, and the deploy reported "pre-warmed and confirmed" while the dropped route's cache partition remained cold. Cloudflare's cache key includes the full URL, so warming the same pathname on another host does not cover that route. The only refusal path required the origin set to be completely empty. Record completeness instead of inferring it: the parser sets hasUnwarmableRoute when any enabled non-workers.dev attachment yields no warmup host (JSONC and TOML, root and env scoped), and the resolved target carries hasUnwarmableProductionRoute. The warm transaction then fails closed: strict mode refuses before warming, and non-strict mode still warms the reachable origins and promotes but reports warmed: false even when every reachable origin confirmed. Disabled routes and workers.dev patterns do not set the flag, and a top-level path-scoped route does not veto an environment whose own routes are all host-wide.
|
Addressed the remaining design hole from the follow-up review in 490f7e6: supported origins can no longer hide unsupported production routes. The resolved target now carries the completeness signal the review proposed, and the transaction fails closed on it. What changed
The previously existing conservative path is unchanged: when no origin is representable, the transaction still refuses via the empty- Deliberately not doneNo route-aware warming (intersecting warm paths against each route's pattern), matching the review's assessment that a route engine is disproportionate here. Failing closed on the completeness bit gives the same guarantee integrity at a fraction of the surface. Verification
|
cdn-warm-deployment.ts imported the runtime formatUnknownError function and the DeployOptions type from deploy.ts, and version-deploy.ts imported the Wrangler CLI helpers from there as well, while deploy.ts imports both modules. The cycle only worked because nothing involved is evaluated eagerly, and it contradicted the layering the transaction extraction was meant to establish: the CLI as a thin caller above the transaction and its adapters. Give the shared pieces homes below every consumer. wrangler-cli.ts owns resolveWranglerBin, buildNodeCliInvocation, validateWranglerEnvName and the WranglerTargetOptions type that previously existed only as ad hoc Pick<DeployOptions, ...> projections; utils/format-unknown-error.ts owns formatUnknownError, replacing a private duplicate in prerender-kv-populate.ts. The warm transaction now defines its own CdnWarmupOptions, and DeployOptions composes the shared types instead of being their source, so imports flow one way: deploy.ts -> cdn-warm-deployment.ts -> version-deploy.ts / target resolver -> wrangler-cli.ts. No behavior change; existing suites cover the moved helpers.
|
Done in 878f30c — the cycle is gone, and the intended dependency direction is now real at both runtime and type level. What changedTwo new bottom-layer modules:
Consumers repointed downward:
The resulting graph is acyclic: VerificationPure refactor — no behavior change intended and none observed: all six affected suites pass unchanged (510 tests; the moved helpers' existing tests now import from |
Replace the hand-rolled TOML lexer (utils/toml.ts) with Wrangler's experimental_readRawConfig, which dispatches on file extension to its real TOML/JSONC grammar. Both formats resolve to the same snake_case shape, so the field-extraction logic is unchanged. wrangler is added as an optional peer dependency so runtime-only consumers are not forced to install it; a malformed config degrades to null via the surrounding try/catch, matching missing-file behavior.
Human note
I looked into this. the bug was that vinext couldn’t tell whether the new Worker version actually handled the warmup request or not. It treated any response 2xx, 3xx response as a success even if it was an old Worker version.
So this PR: stages the uploaded Worker at 0%, prewarms cache keys, then verifies the response using version metadata then promote to 100%
Overview
This fixes #2592 by prewarming the uploaded Worker version before it receives normal traffic. The deploy stages the upload at 0%, requests the real production cache keys with a verified version override, then promotes it to 100%.
Why
Cloudflare Workers Cache includes the invoked Worker version in its cache key by default. That gives each deployed version an isolated cache partition. If an override temporarily misses during deployment propagation, the request can only populate the old version cache; vinext detects the producer mismatch and retries. A timing-based post-promotion barrier cannot provide that structural guarantee.
What changed
cache.cross_version_cacheis enabled, because that deliberately shares cache entries across versions.Validation
vp checkvp run vinext#buildvp run cloudflare#buildThe deploy behavior tests observe the full command and request sequence, version override header, retry on producer mismatch, and absence of promotion after strict failure. TOML and JSONC compatibility remain ordinary parser tests.
References